Certain panoramic images of objects are associated with information relating to the geographic location and orientation from which the image was captured. For example, each pixel of the image may be associated with data that identifies the angle from the geographic location from which the image was captured to the portion of the surface of an object (if any) whose appearance is represented by the visual characteristics of the pixel. Each pixel may also be associated with depth data that identifies the distance from the capture location to the portion of the surface represented by the pixel.
Three-dimensional models of the locations of surfaces appearing in the images may be generated based on the depth data. The models may include polygons whose vertices correspond with the surface locations. The polygons may be textured by projecting visual characteristics of the panoramic image onto the model using ray tracing. A user may select a vantage point from which the models may be displayed to the user.